FUNK, COSINE, AND SINE TRANSFORMS ON 
STIEFEL AND GRASSMANN MANIFOLDS, II 



B. RUBIN 

Abstract. We investigate analytic continuation of the matrix co- 
sine and sine transforms introduced in Part I and depending on a 
complex parameter a. It is shown that the cosine transform cor- 
responding to a = is a constant multiple of the Funk-Radon 
transform in integral geometry for a pair of Stiefel (or Grassmann) 
manifolds. The same case for the sine transform gives the identity 
operator. These results and the relevant composition formula for 
the cosine transforms were established in Part I in the sense of 
distributions. Now we have them pointwise. Some new problems 
are formulated. 



1. Introduction 

Let V n}Tn and V n> k be a pair of Stiefel manifolds of orthonormal frames 
in W 1 of dimensions m and k, respectively; 1 < m, k < n — 1. The 
following integral operators where introduced in 



(<£,*/)(«) = / f(v)[det(v'uu'v)]^ 2 d,v, (1.1) 

= / rtu)[det(v'uu'v)}(«- k » 2 d*u, (1.2) 
Jv n , k 

(S«J)(u) = f f(v)[det(I m -v'uu'v)f a+k -^ 2 d,v, (1.3) 

(S" k <p)(v) = [ ^(u)[det(I m -v'uu'v)f a+k -^ 2 d,u. (1.4) 



Here u G V n> k, v G Vn,mi d*u and d*v stand for the relevant probability 
measures, (•)' denotes the transpose of a matrix (•). 
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We call k f and 5^ k f the cosine transform and the sine transform 

* * 

of /, respectively. Integrals fcV 9 an d S^kV are called the dual cosine 
transform and the dual sine transform. The terminology stems from 
the fact that, in the case k = m = 1, when u and v are unit vectors, 

det (v'uu'v) = (u-v) 2 = cos 2 w, det (I m —v'uu'v) = \ — {u-v) 2 = sin 2 u;, 

where u is the angle between u and v. For integrable functions / and </?, 
integrals fll.lft - fll.4p converge absolutely almost everywhere if and only 
if Re a > m — 1 and represent analytic functions of a in this domain; 
see [3U Theorem 4.2]. 

The present article is a continuation of our previous work [31] devoted 
to the study of operators (jl.ip - fll.4D . The impetus for this research was 
given by the celebrated Matheron's conjecture in stochastic geometry 
[22j p. 189] and a series of related publications in the area of harmonic 
analysis and integral geometry due to Goodey and Howard [H] , Alesker 
and Bernstein [2], Alesker [TJ, Ournycheva and Rubin [2H1I2Z], Zhang 
[4"T] ; see [31] for the detailed exposition of the history and motivation 
of this research. 

In the present paper we suggest an elementary approach to analytic 
continuation of the cosine transform, which enables us to study analytic 
properties of the sine transform and the duals of both transforms. We 
answer a series of questions stated in [31] and related to pointwise 
equalities, the validity of which was proved in Part I only in the sense 
of distributions. 

In section 2 we recall basic definitions and auxiliary facts, which 
are used in the proofs. Main results are presented by Theorems I3.1j 
14.11 15.11 in sections 3,4 and 5, respectively. Section 6 contains some 
consequences, including pointwise inversion of the relevant Funk- Radon 
transform and a composition formula (Theorems 16.11 and 16.41) . We 
conclude with brief discussion of the so-called Cos A - transform, which 
represents a particular case k = m of (II. ip and differs from the latter 
by notation. Such transforms arise in group representations [4"j I2"4"j |2"B"]. 

Acknowledgements. I am thankful to Professors Tomoyuki Kakehi, 
Gestur Olafsson, Angela Pasquale, and Genkai Zhang for useful discus- 
sions. 

2. Preliminaries 

More information about facts presented below can be found in [TBj 
[251 [331 131]. 

2.1. Notation and conventions. Given a square matrix a, \a\ stands 
for the absolute value of det (a). As usual, 0{n) and SO{n) stand for 
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the orthogonal group and the special orthogonal group of IR n , respec- 
tively, with the normalized invariant measure of total mass 1. The 
abbreviation "a.c." denotes analytic continuation. 

Let DJl n , m ~ M. nm be the space of real matrices x = (xij) having n 
rows and m columns; dx = Yl^iTYjLidxij', x' is the transpose of x, 
\x\ m = det(x'x) 1 / 2 , I m is the identity m x m matrix, and stands for 
zero entries. 

Let S m ~ R m ( m + 1 )/ 2 be the space ofmxm real symmetric matrices 
r = (Vjj); dr = Yli<j ^ r i,i- We denote by fl the cone of positive definite 
matrices in S m . The Siegel gamma function of Q is defined by 

„ m— 1 

T m (a)= exp(-ti(r))\r\ a ^ m+1 ^ 2 dr = n m( - m - 1 ^ 4 TT T(a-j/2). (2.1) 

This integral is absolutely convergent if and only if Re a > (m — l)/2, 
and extends meromorphically with the polar set 

{(m-l-j)/2:j = 0,1,2,...}; (2.2) 

see PU], 0, [29 ■ 

For n > m, let V njfn = {d6 9tt n ,m '■ v'v = I m } be the Stiefel manifold 
of orthonormal m-frames in MJ 1 . This is a homogeneous space with 
respect to the action V n>m 3 v — > jv, 7 G 0(n), so that V n>m = 
0(n)/0(n — m). We fix a measure dv on V n)Tn , which is left 0(n)- 
invariant, right 0(m)-invariant, and normalized by 

r 2 m yj- nm /2 

°^] v J v= Tj^Y (2 - 3) 

[23| p. 70]. The notation d*v = a~\ n dv is used for the corresponding 
probability measure. 

We denote by G n ,m the Grassmann manifold of m-dimensional lin- 
ear subspaces £ of R™ equipped with the 0(n)-invariant probability 
measure c?*£. Every right 0(m)-invariant function f(v) on V n , m can 
be identified with a function /(£) by the formula f({v}) = f{v), 
{v\ = vW" 1 G G nm , so that f„ = A/ f(v)d*v. Another 

identification is also possible, namely, / ({f} -1 ) = /(f), {f} 1 " G G n>n - m . 

Lemma 2.1. (The polar decomposition). Let x G 9Jt n ,ra> n > m. If 
rank(a;) = m, then 

x = vr 1/2 , v G V n7m , r = x'x G Q, (2.4) 

and dx = 2- m \r\^ m -^' 2 drdv . 

For this statement see, e.g., [18], [23], [5]. 
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2.2. Zeta integrals. Let S(M n , m ) be the Schwartz space of infinitely 
differentiable functions on 9Jt n>m which are rapidly decreasing together 
with derivatives of all orders. Suppose that n > m > 2 and consider 
the zeta integral 

Z{f,a-n)= J f(x)\x\^ n dx, f e S(m n>m ). (2.5) 
TO 

Lemma 2.2. The integral A2.5\) is absolutely convergent if Re a > m—1 
and extends to Re a < m — 1 as a meromorphic function of a with 
the only poles m — 1, m — 2, . . . . These poles and their orders are 
exactly the same as of the gamma function r m (a/2). The normalized 
zeta integral Z(f,a — n)/T m (a/2) is an entire function of a. 

This statement can be found in [36j [21]; see also J33J Lemma 4.2]. 
The Cay ley- Laplace operator A on the space 9Jl n m of matrices x = (xij) 
is defined by 

A = det(<9'<9). (2.6) 

Here d is an n x m matrix whose entries are partial derivatives d/dxtj. 
More information about this operator, which is neither elliptic nor 
hyperbolic, can be found in [211 E3]- The following identity of the 
Bernstein type holds: 

^\xC 2 "- n = B,{a)\x\ a m -^ (2.7) 

m-l £-1 

Bi, m , n (a) = J] l[{a -i + 2j)(a-n + 2 + 2j + i); (2.8) 

i=0 j=0 

see [33l p. 565]. It allows us to represent meromorphic continuation of 
Z(f,a — n) in the form 

Z(f,a-n) = — --Z(A i f,a+2£-n), Rea> m-l -21, (2.9) 
B t (a) 

£=1,2,... , . 
Lemma 2.3. [331 Theorem 4.4] For / e <S(9Jt n>m ), 

o.c. ; J \ ■ ^ = ; , /(0). (2.10) 

«=o r m (a/2) r m (n/2) ; 1 ; 

The general references related to zeta integrals are fundamental works 
by Bopp and Rubenthaler [3], Igusa [19] . Shintani [38] . 



2.3. The Funk Transform. The classical Funk transform on the unit 
sphere S n ~ 1 C M n is defined by 



(Ff)(u)= / f(v) d u v, ueS"- 1 ; (2.11) 

see, e.g., [HE]. In [34J we introduced the following generalization of 
( 12. lip , in which u G V n> k and v G V n)Tn are elements of the respective 
Stiefel manifolds, 1 < k,m < n — 1. The higher-rank Funk transform 
sends a function / on Km to a function F m k f on V n k by the formula 



(F m ,kf)(u) 

'{»6V„ im : u'v=Q} 

The corresponding dual transform 



f(v) d u v, 



ueVn.k- 



(2.12) 



(Fm,k¥)( v ) = / (p(u)d v u, v<EV n , m , (2.13) 

acts in the opposite direction. The condition u'v = means that sub- 
spaces uR h G G ni k and vM. m G G n ,m are mutually orthogonal. Hence, 
necessarily, 

k + m < n. 

In the case k = m we denote F m = F mjm . 

To give our transforms precise meaning, we set G = 0(n), 



K 
K 



tEG:t 
peG : p 



7 
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7 G 0(n - k) 
5 G Ofn — m 
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Im 


Uq = 


h 







; v = 


Im 


, vo = 






(2.14) 
(2.15) 

(2.16) 



u ,u eV ntk ] v ,v eV n7m . 
Then (12.121) and (I2.13P can be explicitly written as 



{Fm,kf)(u) ■ 
(F m ,k<p)(v) 



f 9u 



V 3 I 9v 

Viz — m, k 



to 







d*u= / f(g u rvo)dT, 



Ao 



d,0 = 



(p(g v pii ) dp, 



(2.17) 
(2.18) 



where g u and gr„ are orthogonal transformations satisfying g u u = u 
and g v vo = v, respectively. 



6 B. RUBIN 

Lemma 2.4. [34"| Lemma 3.2] Let 1 < k, m < n — 1; k + m < n. Then 



(F m>k f)(u)(p(u)d*u= f(v)(F m ,k<p)(v)d*v (2.19) 

provided that at least one of these integrals is finite when f and tp are 
replaced by \ f\ and \ip\, respectively. 

2.4. Cosine and Sine transforms. When dealing with operators 
(11.11) and (11.21) . we restrict our consideration to the case m < k, be- 
cause, if m > k, then \v'uu'v\ = for all v G V ritTn and all u G V n ^. Sim- 
ilarly, for (11. 3j) and ( 11. 4p . we assume m < n — k, because, if m > n — k, 
then 

\I m — v'uu'v\ = \I m — v'Pr{ u }v\ = \v'Pr{ u y±v\ = \v'uu'v\ = (2.20) 

(here u is an arbitrary (n — A;)-frame orthogonal to {u} = uM. k ). The 
case k = n, when v'uu'v = J m , is also not interesting. Clearly, 

(^/)(n) = (Q n _J)(n)= I f{v)\v>uu'v\^ n -Wd*v. (2.21) 

The case k = m, when fc and 5^ k coincide with their duals, is of 
particular importance. In this case we denote 

(M a f)(u)= f f(v)\u'v\ a - m d*v, l<m<n-l, (2.22) 
(Q a f)(u)= [ f(v)\I m -v'uu'v\ {a+m - n)/2 d*v, 2m<n, (2.23) 



V n 



where u G V n ^ m - 



3. Analytic continuation of the cosine transform 

We use the following indirect procedure. Consider the cosine trans- 
form 

(<£.*/)(«)=/" f{v)\i/uu'v\W 2 d.v, ueV n , k , (3.1) 



" 



£ Hi.fe) 7 ^ SO(n). Changing variable 



and set w = 7M , Mo = 
f = jw, we obtain 

(C,J)(7«o)= / f 1 (w)\w'u u' w\ {a ~ k)/2 d,w } 



Vn 



where / 7 (u>) = f(jw). Consider an auxiliary integral 

Fail) = / f\{x(x'xy 1/2 ) \x'uQu' x\ {a ~ k)/2 ilj(x'x) exp(-tr(x'x)) dx, 



where ip(r) is a nonnegative C°° function on the cone Q having a com- 
pact support away from the boundary of Q. Suppose that / belongs 
to C°°(Ki im ). Then the function 

ip~f(x) = fr Y {x{x'x)~ 1 ' 2 ) ip(x'x) exp(— tr(x'x)) (3.2) 

belongs to S(fXH n>m ) and is supported away from the surface \x'x\ = 0. 
Passing to polar coordinates x = wr 1 / 2 , w G V njTn , r G f2 (cf. (12.4j) ). we 
obtain 

F a ( 7 ) = x(a)((S*/)(7tio) ) (3.3) 



x(a) = 2 m a„ m 



\r\ {a - k+n)/2 - d iP(r) exp(-tr(r)) dr, (3.4) 



d = (m+ l)/2. Since x(a) and its reciprocal are entire functions of a, 
which differ from zero for any complex a, then analyticity of C^ k f is 
equivalent to that of F a {^f) and possible poles of both functions are at 
the same points and of the same orders. 

The function F a (j) can be represented as a zeta integral, namely, 

F a (7) = / <p 7 {x)\x'u u' x\ (a - k)/2 dx 

J 9J X n . m 



\y\m k V^(y)dy = Z(^,a- k), (3.5) 

Jm h>m 
where the function 

"iVjdrj (3.6) 



^~^n — k ,m 



belongs to S(wt k ,m); cf. (ET511 . Thus, 

(C, fc /)(7«o) = x(a)- 1 a - *)■ (3.7) 

Theorem 3.1. Letl<m<k<n-l, fe C°°(V n>m ). 

(i) IfRea > m — 1, then the cosine transform (C^ lk f)(u) is represented 
by an absolutely convergent integral. 

(ii) For every u G V n> k, the function a — > (C'^ lk f)(u) extends to the 
domain Re a < m — 1 as a meromorphic function with the only poles 
m — 1, m — 2, . . . . These poles and their orders are exactly the same 
as of the gamma function T m (a/2). 

(iii) The meromorphic continuation of (C^ lk f)(u) can be represented in 
the form 

(C, */)(t«o) = — n Z(A '^' a + 2£ ~ k) > (3 - 8) 

Rea> m-1-21 £=1,2 
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where Bi >m ^{a) is the Bernstein polynomial 

m-l t-i 

Be, m , k (a) = } [ Y[(a - i + 2j)(a - k + 2 + 2j + i) (3.9) 



i=0 j=0 



and <y5 7 is defined by h3.b}) - [3Jfy . 

(iv) The normalized integral (C^ lk f)(u)/T m (a/2) is an entire function 
of a and belongs to C°°(V n ,k) in the u-variable. 

(v) If, moreover, k + m < n, then 



(^m,kf)( U ) /T -, -w n 

«=o r m (a/2) 



(3.10) 



where 



r m (n/2) 



r m (/c/2) r m ((n - fc)/2) 
and (F mt kf)(u) is the Funk transform ( 2.17\ ). 

Proof. Statements (i) - (iv) are immediate consequences of Lemma [21 
and equality (12. 9p . In particular, smoothness of the analytic continua- 
tion 

a.c.{C^ k f){u)/T m {a/2) (3.11) 

in the u- variable follows from (13. 8p . because A^ 7 is a smooth function 
of 7 G SO (n). To prove (v), by (13. 7p we have 



a.c. 



(<£,*/)(«) 



a.c. 



1 Z(<p~f, a — k) 



a=o r m (a/2) a=o x(a) r m (a/2) 
Hence, by ( 12TPU]) . (13761) . and (IQl . 



(3.12) 



a=6 r m (a/2) *r(0)r m (fc/2) 

_fcm/2 



x(o)r m (fc/2) 



77 




^7(0) 

i r l' r i)~ 1 ^ 2 ) ^(v'v) exp(— tr(n'n)) dn. 



Since n — fc > m, we can use Lemma 12.11 to pass to polar coordinates 
and get an expression of the form CI1I2, where 



to 




7T 



km/2 



0~n—k,r 



n 



r ^n-k)/2-d^ exp (_ tr ( r )) dr ^ 

r m (n/2) 1 



r m (fc/2) / 2 r m (fc/2) r m ((n - fc)/2) / 2 



(use (12.31) ). Integral I\ is exactly the Funk transform (12 . 1 Tf) . This 
proves flHTTOl) . □ 

Some comments are in order. 

1. Equality (I3.10P was obtained in [31] in the weak sense, using the 
Fourier transform technique; cf. formula (6.9) in that paper. Our new 
proof is much simpler and more informative. 

2. The particular case k = m in Theorem 13.11 characterizes analytic 
properties of the operator M a f defined by (12.221) . For example, (I3.10p 
yields 

Q " C n T^V = C m,m(F m f)(u), (3.13) 



where 

r m (n/2) 



™ r m (m/2)r m ((n-m)/2)' 

and 



2m < n, 



(F m f)(u)= f(v)d u v, ueV n , m} (3.14) 

' {veV n>m :u'v=0} 



is the Funk transform. 

3. The particular case / = 1, when 



1 T m (A;/2) T m {{a-k+n)/2) V ; 



(cf. (8.16) in [34]) illustrates the role of the gamma function T m (a/2) 
in the statement (ii) of Theorem 13.11 

4. It is known [25| 133] that analytic continuation of the zeta integral 
(12. 5 p at the so-called Wallah set a — 0, 1, 2, ... m — 1 is represented as a 
convolution with a positive measure, which can be explicitly evaluated. 
We conjecture, that a similar result holds for the normalized cosine 
transform (C^ lk f)(u)/T m (a/2) and the corresponding measure can be 
explicitly evaluated in the form, which does not contain the auxiliary 
function ip (so far, we succeeded in doing that only for a = 0). 

4. Analytic continuation of the sine transform 

The sine transform of a function f{v) on V n ^ m is a function fc /)(u) 
defined by 

(S^ k f){u)= I f(v)\I m -v'uu'v\^ +k ^ 2 d^ ueV n , k . (4.1) 

We assume k + m < n, because otherwise \I m — v'uu'v\ = for all 
v G V n>m and M G V^; see (I2.20p . 



10 



B. RUBIN 



Theorem 4.1. Let 1 < k,m < n - 1, k + m < n, f G C°°{V ntm ). 

(i) If Rea > m — 1, then the sine transform {S^ lk f){u) is represented 
by an absolutely convergent integral. 

(ii) For every u G V n ^, the function a — > {S^ lk f){u) extends to the 
domain Re a < m — 1 as a meromorphic function with the only poles 
m — l,m — 2, . . . . These poles and their orders are exactly the same 
as of the gamma function T m {a/2). 

(iii) The normalized integral (S^ lk f)(u)/T m (a/2) is an entire function 
of a and belongs to C°°(V n) k) in the u-variable. 

(iv) In the case k = m, 2m < n, when S!^ k f = Q a f is the integral 
H2.23\) . we have 



a.c. 



(Q a f)(u) 



r m (n/2) 



a=6 r m (a/2) r m (m/2)r m ((n-m)/2) 
provided that f is right 0(m) -invariant. 
Proof. We have 

\I m - v'uu'v\ = \I m - v'Pr {u} v\ = \v'Pr {u} ±v\ = \ 



/(«) 



(4.2) 



v'uu'vl 



0. 



where u is an arbitrary (n — fc)-frame orthogonal to {u} = uMr. Hence, 



f(v) \v'm'vt +k - n ^d^ = {C- k f){u). (4.3) 



It remains to apply Theorem 13.11 To prove (14. 2 p , we make use of (12 .2 1[) 
with k = m. Then, as above, 



a.c 



(Q a f)(u)) 



a.c. 



(C m n - m f)(u) 



c=o r m (a/2) a=d r m (a/2) 



chh 



where 
h = 



f-, 



to 




d*u 



f 7 







O(m) 



r |m/2 d^( r ) exp(-tr(r)) dr, c 



u) du = f 
T m (n/2) 







Here 7 is a rotation that takes 







T m (m/2)T m ((n - m)/2) I 2 
to u G {m}^. Hence, 7 takes 







to a certain frame, say, Mi, that spans the same subspace as u. 
Since / is right 0(m)-invariant, then Ii = f(u), and we are done. □ 
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5. Analytic continuation of the dual cosine transform 
The dual cosine transform of a function (p on V n> k is denned by 

(C° k <p)(v)= f ip(u)\v'uu'v\^ k ^ 2 d,u } veV n , m , m<k. (5.1) 
Jv n>k 

We will be dealing with smooth functions ip G C°°(V n ^). Moreover, 
since (p(u) and (f(wy) V7 G O(k) have the same dual cosine transform, 
we can assume that ip is right 0{k) -invariant. We denote by u G V n) n-k 
and v G V n , n - m arbitrary frames orthogonal to subspaces {u} and {v}, 
respectively. Then 

\v'uu'v\ = \I m — v'uu'v\ = \I n -k — u'vv'u\ = \I n -k — u'Pti v \u\ 

= |£t / Pr/ t ,i_L'u| = \uvv'u\. 

Setting <fi(u) = <p(u), we obtain 

= I vi{u)\u'vv'u\^' 2 d*u (5.2) 

* 

Thus, analytic properties of C^fcV 9 can be derived from Theorem 13.11 

Theorem 5.1. Let l<m<k<n — 1 and let ip be a right O(k)- 
invariant function in C°°(V njk ). 

* 

(i) If Re a > m — 1, then dual cosine transform (Cmk ( f)( v ) ^ s repre- 
sented by an absolutely convergent integral. 

* 

(ii) For every v G V n ,m, the function a — > {Cmk l P)( v )) extends to the 
domain Re a < m — 1 as a meromorphic function with the only poles 
m — 1, m — 2, . . . . These poles and their orders are exactly the same 
as of the gamma function T n _ k ((a + n — k — m)/2). 

(iii) The normalized integral (C^ k (p)(v)/T n _k((ot + n — k — m)/2) is an 
entire function of a and belongs to C°°(V^ jm ) in the v-variable. 

(iv) If k + m < n, then (C° k (p)(v)/T m (a/2) extends as a meromorphic 
function with the only possible poles — 1, —2, .... Moreover, for every 

V G V n ,m, 

ax n r 1 /o\ = Ck > m [V) , 5.3 

«=o T m (a/2) 

_ r m (n/2) 



where 



T m (k/2) V m {{n-k)/2) 

* 1^—^—1 

and (F m i^)(«) is the dual Funk transform $2. 13\) . 
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(v) If k + m > n, then for every v G V n ,m, 

a-c- p j7—- r r^r = c fc , m F n _ fci „_ m ^! w , 5.4 

a=fc+m-n 1 n _ fc ((Q: + 71 — K — 771) / 2) 

where 

T n - k (n/2) 



Cjfc. 



r„- fc (m/2) r n _ fc ((n - m)/2) 
and (F n _fc )n _ m <y9i)({;) zs £/ie relevant Funk transform; cf. {2.11 ). 

Proof. Owing to (15.21) . statements (i)-(iii) and (v) are immediate con- 
sequences of the respective statements in Theorem 13. 11 To prove (iv), 
we observe that by (12.11) . 

r n _ fc ((a + n — k — m)/2) = c{a) F m (a/2), (5.5) 

where c(a) = 7r( n ~*~ m ) m / 2 r n _fc_ m ((a;-|-n — k — m)/2) is a meromorphic 
function with the polar set {—1, —2, . . . }. Denote 

A ( ,_ fet#) _ c(a)(C^ k <p)(v) 

r ra (a/2) -r„_ fc ((a + n-A;-m)/2)' Wea>m X ' 

By (iii), this function extends analytically to Re a > — 1 and the an- 
alytic continuation belongs to C°°(V^ im ). Now for any test function 
u G C°°(V nim ), owing to fl3~T0|) and ftZJ9\f . we have 



a.c. A a , u) = a.c. I <p, — — ' 1 = I (p, a.c. 



^ a =o °" ' «=d \'Y m {aj2)) V. «=» T m (a/2) 

and (15.31) follows. □ 

6. Some consequences 
6.1. Inversion of the Funk transform. 

Theorem 6.1. Let (p = F m ^f , where f is a C°° right 0(m) -invariant 
function on V n , m , l<m<k<n — m. Then, for every v G V n ,m, 

{C%, h <p)(v) r m (n/2) 
?- c - r i W\ = C /W> c= (6.1) 

a=fc+m-n 1 m [Oi/2) L m (k / 2) L m {m / 2) 

Proof. By j3U Theorem 4.3], if -Re a > m — 1, then 

(C" k <p)(v) r m ((n - m)/2) (Q a+ "- fc - m /)(t;) 



r m (a/2) r m (fc/2) r m ((a + n-A;-m)/2)' 



(6.2) 
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where Q^+n-k-mj ig the 

sine transform from Section HI By Theorem 
14. 14 the right-hand side of (I6.2p extends as an entire function of a. Now, 
taking analytic continuation and using (14.21) . we obtain the result. □ 

Remark 6.2. It is interesting to note that in general, the function 
a — > (C mk (f)(v)/T m (a/2) admits poles —1, —2, . . . ; see Theorem 15.11 

(iv) . However, as (16. 2h reveals, these poles disappear if C m k acts on 
the image of the Funk transform F m ^. 

Remark 6.3. Theorem 16.11 can be reformulated in the language of 
Grassmannians. The reader can easily do this, using connection for- 
mulas from [SU Section 3.2]. A series of inversion results for the Funk- 
Radon transform on Grassmannians can be found in the fundamental 
works by Gelfand and his collaborators [HI Bj, Grinberg |15j . Kakehi 
[20] ; see also a pioneering paper by Petrov [20], Grinberg and Rubin 
[T6] , and Zhang [40]. Diverse problems of integral geometry related to 
Grassmann manifolds were studied in [BJ HH [T21 [131 El E2l [37] . Our 
Theorem 16.11 has a completely different flavor, agrees with the known 
case m = 1 (cf. formula (1.13) in [30]) and sheds new light to this circle 
of problems. 

6.2. A composition formula; the case k = m. In this particular 

case the cosine transform (M a f)(u) = (C^ m /)(-u), u G V n;m , has a 
number of important features. We normalize it by setting 

Wf){u) = 5 n , m (a)(M a f)(u) 

= Sn,m( a ) [ f(v)\u'v\ a - m d*V, (6.3) 

r m (m/2) r m ((m-q)/2) . 

o n ,m{ot) = . , , , , . , a N = {1,2, ...}. 

r m (n/2) T m (a/2) 

The excluded values of a form the polar set of T m ((m — a)/2). Integral 
(16. 3 p converges absolutely for any integrable function / when Re a > 
771—1. If / G C°°(V^ im ), then, by Theorem 13. 1] analytic continuation of 
a function a — » (^ a f)(u) is well-defined for all complex a ^ 1,2,... 
and belongs to C°°(V^ jm ). We denote 

(^C.c./X") = a.c (.^ a f)(u). (6.4) 

Theorem 6.4. Let / G C°°(V^, )m ) fre a n'<?/i£ 0(m) -invariant function 
on V n ,m, 2m <n. If a, 2m — a — n ^ N, then for every u G V n ,m, 



2m— a— n 
a.c. 



^U){u) = f{u). (6.5) 
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Proof. In [3U Theorem 6.4] we proved that if Re a > m — 1, a ^ 
m, m + 1, m + 2, . . ., then 

a.c. {^^ a f,u) = (f,u), (6.6) 

/3=2m— a— n 

for any test function uj G C°°(V ntm ). Since / G C°°(V^ i?n ), then, by 
Theorem 13. 1[ analytic continuation of a function f3 — > f)(u) is 

well-defined for all /3 ^ 1, 2, . . . and represents a C°° function on V^ >m . 
Hence, 

{^ a _ c yg a f,oj) = (a.c.^^ /,^) = a.c.(^ ^ a f,Lu), 
and the aforementioned result from [33] yields ( t y#^~ a ~" t y# a /, w) = 



(f,oj). Since, by Theorem 13. 11 the function a c ^ a f)(u) is smooth, 
then we have a pointwise equality 

(J?l m c : a - n ^ a f)(u) = f(u), (6.7) 

provided that 

Re a > m — 1, a ^ to, to + 1, m + 2, . . . . 

To extend this result to all complex a such that a, 2m — a — n ^ N, 
we observe that (^a"c7 a ~ n ^a.c.f)( u ) * s separately analytic in a and 
(3 in the domain A = {(a, (3) G C 2 : a N, /3 ^ N}. Hence, by 
the fundamental Hartogs theorem [35], this function is analytic in z = 
(a, f3) G A. Now (16.51) follows from (16.71) by the uniqueness of analytic 
continuation. □ 

6.3. The Cos A -transforms. Transformation (12.221) can be found in 
the literature under different names (or without naming) and with dif- 
ferent notation. For instance, let 

(Cos x f)(u) = ! f(v)\u'v\ x - p d*v, (6.8) 

ReX>p-l, p = n/2, u G V n<m ; 
cf. [23], where a similar operator is presented in a slightly different 
form. Following [24J, we call (I6.8P the Cos A -transform of /. The con- 
nection between (16.81) and (12.221) is 

Cos A / = M x+m ~ p f. (6.9) 

In terms of (I6.8P formula (13.131) becomes 

>m F m f. (6.10) 

A=p- m r m ((A + m — p)/2) 
The composition formula (16.51) transforms to 

Cos- x Cos x f = f, ±\ + m-p=£ 1,2,... , (6.11) 
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where Cos x denotes analytic continuation of the normalized integral 
(Cos x f)(u) = ~6 n<m (X) j f(v)\u'v\ x - p d*v. (6.12) 

T m (m/2) T m ((p-A)/2) 



S n ,m{X) 



T m (p) r m ((A + m-p)/2)' 



6.3.1. The case m — 1. This case deserves special mentioning. The 
corresponding cosine transforms on the unit sphere S 1 ™ -1 in IR n are 
well-known in analysis. We recall some results; see a survey article [31] 
and references therein. In the "A-notation" the normalized "rank-one" 
Cos A -transform is defined as analytic continuation of the integral 

(c ° sA/)(u) = mm-p+w) U M '"'"'^ (613) 

where p = n/2 and d*v is the probability measure on S 1 ™ -1 . Let 
{Xi,v{v)} be an orthonormal basis of spherical harmonics on 5* n_1 . Here 
j = 0, 1, 2, . . . , and u — 1,2, ... , d n (j), where d n (J) is the dimension of 
the subspace of spherical harmonics of degree j. If 

/ V /,./ f jtk = [ f(v)Y jtk (v)dv, (6.14) 

(the Fourier-Laplace expansion of /), then 

Cos x f = J2 c ^kY j)k , (6.15) 

where 

9,a= ( lj r(( J+ p + A)/2) ^ lseven ' (6.16) 
[ if j is odd, 

Let L PtE = L p ^(S n ~ 1 ) be the Lebesgue space of even p-integrable 
functions on S n ~ 1 . The asymptotic of as j — > oo implies the follow- 
ing embeddings of the range Cos x (L P:e ), 1 < p < oo, into the relevant 
Sobolev spaces LJ . 

Theorem 6.5. (cf. Corollary 3.3 in [31]) Let 

1 1 



7± = Re A ± 



(n-1) 



p 2 

A £ {p,p + 2,p + 4,...}U{-p-l,-p-3,-p-5,...}. 
T/ie following proper embeddings hold: 

LJ- e C Co S A (L p , e ) C (6.17) 
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If p = 2, then 



Cos\L 2)e ) = Lf A 



7 2,e 



It is a challenging problem to possibly extend these results to the 
higher-rank case m > 1. 
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